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Abstract. For every genuine irreducible admissible smooth representation tt of 
the metaplectic group Sp(2n) over a p-adic field, and every smooth oscillator rep- 
resentation oj^p of Sp(2n), we prove that the tensor product it <g) is multiplicity 
free as a smooth representation of the symplectic group Sp(2n). Similar results are 
proved for general linear groups and unitary groups. 



1. Introduction and main results 

Fix a non-archimedean local field k of characteristic zero. The following multi- 
plicity one theorem for general linear groups, unitary groups and orthogonal groups, 
which has been expected since 1980's, is established recently by Aizenbud-Gourevitch- 
Rallis-Schiffmann in |AGRS07| . 

Theorem A. Let G denote the group GL(n) ; U(n), or 0(n), define overk, and let 
G' denote GL(n — 1), U(n — 1), or 0(n — 1), respectively, regarded as a subgroup of 
G as usual. Then for any irreducible admissible smooth representation n of G, and 
7r' of G' , one has that 

dim Home (7r (g) 7r', C) < 1. 

As will be clear later, the groups GL(n), U(n) and O(n) are automorphism groups 
of "Hermitian modules". Therefore, we consider Theorem A the multiplicity one 
theorem in the "Hermitian case". It is the first step towards the famous Gross- 
Pras ad Conje cture ( [GP921 IGP941 lGR06l IGGP08] ) . 

In |GGP08j . W. T. Gan, B. Gross and D. Prasad formulate an analog of the Gross- 
Prasad Conjecture in the "skew- Hermitian case" . The corresponding multiplicity one 
theorem, whose proof is the main goal of this paper, is the following: 

Theorem B. Let G denote the group GL(n) 7 U(n), or Sp(2n), defined over k, and 
regarded as a subgroup of the symplectic group Sp(2n) as usual. Let G be the double 
cover of G induced by the metaplectic cover Sp(2n) ofSp(2n). Denote by the 
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smooth oscillator representation o/Sp(2n) corresponding to a non-trivial character 
ip ofk. Then for any irreducible admissible smooth representation tt of G, and any 
genuine irreducible admissible smooth representation n' of G, one has that 

dim Horned (g) tt' <g) u^, C) < 1. 

Recall that an irreducible admissible smooth representation of G is said to be 
genuine if it does not descent to a representation of G. For symplectic groups, 
Theorem B is conjectured by D. Prasad in |Pr96[ Page 20]. 

The "Horn" -spaces in Theorem A and Theorem B are extreme cases of their very 
important generalizations, namely, Bessel models and Fourier- Jacobi models, re- 
spectively. For definitions of these models, see |GGP08[ Part 3] , for example. As ex- 
plained in [GPSR97J, uniqueness of Bessel models is the basic starting point to study 
L-functions for orthogonal groups by the Rankin- Selberg method. Similarly, unique- 
ness of Fourier- Jacobi models is basic to study L-functions for symplectic groups and 
metaplectic groups ([GJRS09J). The importance of Theorem A and Theorem B lies 
in the fact that they imply uniqueness of these models in general (c.f., |GGP08t Part 
3], |GJRS09l Theorem 4.1] and |JSZ09j ). Various special cases of these uniqueness 
are obtained in the literature (see |Nov76l IGPSR871 IBFG921 IGRS981 IBROOl IGGP08] 
for example). 

In order to prove our main results uniformly, we introduce the following notation. 
By a commutative involutive algebra, we mean a finite product of finite field exten- 
sions of k, equipped with a k-algebra involution on it. Let (A, r) be a commutative 
involutive algebra. Let E be an A-module which is finite-dimensional as a k-vector 
space. For e = ±1, recall that a k-bilinear map 

(,) E : E x E ^ A 

is called an e-Hermitian form if it satisfies 

(u,v)e = c(v,u) T E , (ou,v)e = cl(u,v)e, aeA,u,veE. 

Assume that E is an e-Hermitian A-module, namely it is equipped with a non- 
degenerate e-Hermitian form (,)e- Denote by U(E) the group of all A-module 
automorphisms of E which preserve the form (,) E - Depending on e = 1 or — 1, 
it is a finite product of general linear groups, unitary groups, and orthogonal or 
symplectic groups (see |SZ082t Section 3] for more details). 

View A 2 as a standard hyperbolic plane, i.e., it is equipped with the e-Hermitian 
form ( , )a 2 so that both e\ and e2 are isotropic, and that 



(ei,e 2 ) A 2 = 1, 
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where e 1: e 2 is the standard basis of A 2 . The orthogonal direct sum E®A 2 is again an 
e-Hermitian A-module. Note that U(E') is identified with the subgroup of U(E(BA 2 ) 
fixing both e\ and ei- Denote by 3(E) the subgroup of XJ(E © A 2 ) fixing e\. 

To be explicit, view E © A 2 as the space of column vectors of A © E © A, then 
3(E) consists of all matrices of the form 



j(x,u,t) :-- 



2 

Ox U 



1 — U T X t 

x 
I 



where 



x e U(E), ueE, te A T= ~ e ■= {t e A \ t T = -et}, 

and u T is the map 

E -> A, v !->■ (v,u) E - 
The unipotent radical of J(E) is 

H(£) := {j(l,u,t) \ueE,te A T= - £ } = Ex A T= ~\ 
with multiplication 

x/ / ix / / / (u,u')e (u',u)es 
(u, t)(u', t') = (u + u',t + t'+ - 

By identifying j(x,u,t) with (x, (u, i)), we have 

J(S) = XJ(E) x H(E). 



We will see that Theorem B is a consequence of the following theorem in the 
skew- Hermit ian case, namely when e = — 1. 

Theorem C. For every irreducible admissible smooth representation 7Tj of 3(E), and 
every irreducible admissible smooth representation 7i\j ofXJ(E), one has that 

dimHom U ( £ )(7Tj,7r u ) < I. 



To prove Theorem C by the method of Gelfand-Kazhdan, we extend V(E) to 
a larger group, which is denoted by IJ(E), and is defined to be the subgroup of 
Aut(E) x {±1} consists of pairs (g,S) such that either 

5 = 1 and (gu, gv) e = (v,u) e, u,v G E, 

or 

5 = — 1 and (gu, gv) e = (v,u) e, u,v&E. 
Here Aut(E') is the automorphism group of the underlying additive group of E. Note 
that for every element (g, S) G tJ(E), if 5 — 1, then g is automatically A-linear, and if 
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5 = — 1, then g is r-conjugate linear. This extended group is first introduced implic- 
itly by Moeglin-Vigneras-Waldspurger in [M VW871 Proposition 4.1.2]. It contains 
13(E) as a subgroup of index two. Let 13(E) act on K(E) as group automorphisms 
by 

(1) (g,8).(u,t):=(gu,8t). 

This extends the adjoint action of 13(E) on H(E). The semidirect product 

3(E) := X3(E) k R(E) 

contains 

J(£) = XJ(E) x H(£) 

as a subgroup of index two. 

The main body of this paper is devoted to a proof of the following 

Theorem D. Let f be a generalized function on 3(E). If it is invariant under the 
adjoint action ofU(E), i.e., 

f(9jg- l ) = f(j), for all gE 13(E), 

then 

fCgj^g' 1 ) = fU), for all g G JJ(E) \ \J(E). 

The usual notion of generalized functions will be explained in the next section. 
As we will see in the proof, when 13(E) is GL(n), U(n), or 0(n), Theorem D is an 
easy consequence of [AGRS07, Theorem 2 and Theorem 2'], and even when 13(E) is 
Sp(2n), the proof of Theorem D depends on the cases of general linear groups and 
unitary groups. 

The author thanks Gerrit van Dijk, Dipendra Prasad, Lei Zhang and Chen-Bo Zhu 
for helpful comments. He is grateful to Dihua Jiang for teaching him the method of 
Gelfand-Kazhdan. 

2. Reduction to the null cone 

We first recall some basic notions and facts about distributions and generalized 
functions. By a t.d. space, we mean a topological space which is Hausdorff, secondly 
countable, locally compact and totally disconnected. By a t.d. group, we mean a 
topological group whose underlying topological space is a t.d. space. For a t.d space 
M, denote by C^(M) the space of compactly supported, locally constant (complex 
valued) functions on M. Denote by D~°°(M) the space of linear functionals on 
C^(M). Such functionals are called distributions on M. If M is furthermore a 
locally analytic k-manifold or a t.d. group (see |Sc08t Part II] for the notion of 
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locally analytic manifolds), denote by D^(M) the space of compactly supported 
distributions on M which are locally scalar multiples of Haar measures. In this case, 
denote by C~°°(M) the space of linear functionals on D™(M). Such functionals are 
called generalized functions on M. 

Let tp : M — > N be a continuous map of t.d. spaces. If it is proper, then we define 
the push forward map 

tp* : D~°°(M) ->■ D-°°(N) 

as usual. Recall that tp is said to be proper if </? _1 (C) is compact for every compact 
subset C of N. When tp is a closed embedding, tp* identifies D~°°(M) with distri- 
butions in D~°°(N) which are supported in (p(M). If tp is not proper, then the push 
forward map is still defined, but only for distributions with compact support. 

If tp : M — > N is a submersion of locally analytic k-manifolds, then the push 
forward map sends D^'(M) into Djf(N), and its transpose defines the pull back map 

p* : cr°°(iv) ->■ cr°°(M). 

When tp is a surjective submersion, tp* is injective. 

If G is an (abstract) group acting continuously on a t.d. space M, then for any 
group homomorphism xg '■ G — ?■ C x , put 

D-^(M) := {co e D~°°(M) \ (T g )*uj = XG (g)u, 9 e G}, 

where T g : M — > M is the map given by the action of g e G. If furthermore M is a 
locally analytic k-manifold, and the action of G on it is also locally analytic, denote 
by 

C^°(M) c cr°°(M) 

the subspace consisting of all / which are XG-equivariant, i.e., 

f(g-x) = Xa{g)f{x), for all g G G, 

or to be precise, 

T;(f) = XG (g)f, forall^GG. 

Now we return to the notation of the last section. Recall that E is an e-Hermitian 
A-module. Denote by 

X e ■ U(£) ^ {±1}, (g,6)^6 
the quadratic character projecting to the second factor. Let \J(E) act on 3(E) by 

(2) g.j-^gjx^g- 1 - 
Then Theorem D is equivalent to saying that 

(3) C£°(J(£)) = 0. 
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Denote by qIa{E) the (associative) algebra of all A-module endomorphisms of E. 
The involution r on A extends to an anti- involution on gl^(E), which is still denoted 
by r, by requiring that 

(x T u, v )e = (u, xv )e, u,v E E. 

Write 

u(E) := {x e Ql A (E) I x T + x = 0}. 

This is the Lie algebra of (the k-linear algebraic group) U(i?). Let U(-E) act on u(E) 
and E by 

| (g,5).x := Sgxg" 1 , xeu(E), 
\ (g,S).u := 5gu, 116 E, 

and act on u(E) x E diagonally. The infinitesimal analog of ([3]) which we need is 

(5) C^°>(£) xE) = 0. 

Write 

XJ(A) := {a G A | aa T = 1}, 
and denote by Z(i?) the image of the map 

XJ(A) V(E) 

given by scalar multiplications. Note that Z(i?) coincides with the center of U(E). 
The Lie algebra of U{A) is 

u(A) := {a E A \ a + a T = 0}, 
and the Lie algebra %{E) of Z(i?) is the image of the map 

u{A) u{E) 

given by scalar multiplications. Note that l{E) may not coincide with the center of 
u(E). Denote by He the set of unipotent elements in U(£ l ), and by Me the set of 
nilpotent (as k-linear operators on E) elements in u(-E'). 

If the commutative involutive algebra A is simple, i.e., if it is a field or a product 
of two isomorphic fields which are exchanged by r, then E is automatically a free 
A-module of finite rank. In this case, put 

sdim(i?) := max{rank J 4(i?) — 1, 0}. 

In general, A is uniquely of the form 

A = Ai x A 2 x • • • x A n , 
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where n > 0, and Ai is a simple commutative involutive algebra, i = 1,2, ••• ,n. 
Note that A^ ® A E is canonically an e-Hermitian Aj-module. Put 

n 

sdim(E') := sdim(Aj ® A E). 

i=l 

We prove the following induction result by Harish-Chandra descent in the remain- 
ing part of this section. 

Proposition 2.1. Assume that for all commutative involutive algebras A° and all 
e-Hermitian A° -modules E° , 

(6) sdim(£°) < sdim(E) implies (J(£°)) = 0. 

Then every f G C£~(J(£)) is supported in (Z(E)U E ) k E(E). 

The infinitesimal analog is 

Proposition 2.2. Assume that for all commutative involutive algebras A° and all 
e-Hermitian A° -modules E° , 

sdim(£°) < sdim(E) implies C^(u(£°) x E°) = 0. 

Then every f G C^(u(E) x E) is supported in {z{E) + Me) X E. 

The proof of Proposition 12.11 is close to that of |SZ082l Proposition 5.2]. Without 
loss of generality, we assume that E is faithful as an A-module. For every semisimple 
element x G U(J5), denote by A x the subalgebra of qIa(E) generated by x, x T and 
A. Then (A x ,t) is again a commutative involutive algebra. Define an A-linear map 
^ t a x /a '■ A x — > A by requiring that the diagram 

A x trA * M ) A 



4 „ a tT (A ® A A x )/A 

A ® A A X ► A Q 

commutes for every quotient field Aq of A, where the bottom arrow is the usual 
trace map for a finite dimensional commutative algebra over a field. Write E x := E, 
viewed as an /^-module. 

Lemma 2.3. There is a unique e-Hermitian form (, )e x on the A x -module E x such 
that the diagram 



E T x E T A, 



tr A x /A 



Ex E a 
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commutes. 

Proof. Uniqueness is clear. For existence, we define the form by requiring that 

tr Ax/k (a(u, v) Ex ) = tr A/k ({au, v) E ), a G A x , u,v G E. 
Then all the desired properties are routine to check. □ 

Now E x is an e-Hermitian A x -module. Note that \J(E X ) is a subgroup of IJ{E), 
and the homomorphism 

& : 3(E X ) = XJ(E X ) x (E x x Al=-*) -> J(£) = U(£) K (E x ^=" e ), 

(y, (u,t)) i ^ (2/,(tt,trA./A(*)) 

is U(£' x )-intertwining. 

We omit the proof of the following elementary lemma. 

Lemma 2.4. With the notation as above. If x ^ ^{E), then 

sdim^a;) < sdim(.E'). 

Now assume that x £ 'Zi(E). For any 

j = (y,h)EJ(E x ) = U(E x )KR(E t ), 

denote by J(j) the determinant of the k-linear map 

1 - Ad,-! : u(E)/u(E x ) -> u(£)/u(£ x ). 

Note that Ad y preserves a non-degenerate k-quadratic form on u(E)/u(E x ), which 
implies that J is \J(E X ) -invariant. Put 

J(£ x )° := {j G J(E X ) | J(j) ^ 0}. 

It contains the set xUe x x H(E x ). One easily checks that the map 

Px :ij(E)xJ(E x )° -> J(£), 

is a submersion, and we have a well defined map (c.f. [JSZ08J Lemma 2.5]) 

(7) r x :^(m)^Cr^(J(E x n 
which is specified by the rule 

pl(f) = XE®r x (f), / e C£°(J(JE0). 

Lemma 12.41 and the assumption (J6]) easily imply the vanishing of the range space of 
([7]) (c.f. [JSZ081 Lemma 2.6]). Thus every / G C^°(J(i?)) vanishes on the image of 
p x . As x is arbitrary, we finish the proof of Proposition 12. II The proof of Proposition 
12.21 is similar. We omit the details. 
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With the idea of linearlization by Jaquet-Rallis ([JR96]) in mind, the goal of this 
section is to prove the following 

Proposition 3.1. Assume that for all commutative involutive algebras A° and all 
e-Hermitian A° -modules E° , we have 

Then C^(3(E)) = 0. 
The Lie algebra of H(E) is 

t)(E) := Ex A T= ~ e 

with Lie bracket given by 

[(it, t), (u, t')) := (0, (u, u') E - («', u) E ). 
The Lie algebra of 3(E) is 

j(E):=u(E)x[)(E), 
where the semidirect product is defined by the Lie algebra action 

x.(u, t) := (xu, 0). 
Let U(-B') act on )(E) by the differential of its action on 3(E), i.e., 

9-3 :=XE(g)Ad s (j), jej(E). 
It is easy to see that as a U(i?)-space, 

)(E) =u(E) xEx A T= ~ e , 

where A T= ~ e carries the trivial U(i±7)-action. Recall the following localization princi- 
ple which is due to Bernstein. See [B e84[ section 1.4] or [AGRS07, Corollary 2.1]. 

Lemma 3.2. Let ip : M — >■ be a continuous map of t.d. spaces, and let G be 
a group acting continuously on M preserving the fibers of (p. Then for any group 
homomorphism xg '■ G — )■ C x , the condition 

D-™(<p- l (x)) = for allxeN 

implies that 

D-^(M) = 0. 

In particular, the localization principle implies the following 
Lemma 3.3. If 

D-™(u(E) x E) = 0, 

Then D-™()(E)) = 0. 
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We need the following obvious fact of exponential maps in the theory of linear 
algebraic groups. 

Lemma 3.4. The set of unipotent elements in J(E) is Ue X ~R{E), the set of al- 
gebraically nilpotent elements in j(E) is Me x ty{E), and the exponential map is a 
XJ (E) -intertwining homeomorphism from Me x fy{E) onto Ue x H(£J). 

In all cases that concern us, whenever M is a locally analytic k-manifold with a 
locally analytic U(.E)-action, there is always a canonical choice (up to a scalar) of a 
positive smooth invariant measure on M. Therefore the space C^°(M) is canonically 
identified with D~™(M). We will use this observation freely. 

For every z G Z(E), it is clear that zUe x H(i£) is U(J5)-stable. 

Lemma 3.5. // 

(8) C^W) >< *0 = 0, 

then 

D-™{U E x U(E)) = 0. 



Proof. By Lemma [3.31 we have that 

^°°0(^)) = o, 

which implies that 

x (,(£)) = 0. 

The lemma then follows from Lemma 13.41 □ 
Lemma 3.6. If 

(9) ^(u(E) x E) = 0, 

D-™(Z(E)U E x H(£)) = 0. 

Proof. It is easy to see (by using the trace map) that the map 

Z( £ )« E KH(£) - Z(£), 

(za;,/i) i — y z, v ' 

is a well defined continuous map. By the localization principle, it suffices to show 
that 

(10) D xT( zU e x K(E)) = 0, for all z G Z(E). 
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Given z G Z(i?), denote by T z the left multiplication by z. One easily checks that 
the diagram 



U E x tt(E) 

a 

U E x H(E) 

commutes for all g G U{E), and the diagram 



^ X H(£) 
-»- «W B x H(£) 



W E x H(£) z^fe k H(£) 



zW E x H(£) 



W s x H(£) - 

commutes for all g G U(-E') \ U(£ l ), where all vertical arrows are given by the actions 
of the indicated elements. Therefore ( flOl) is a consequence of Lemma 13.51 □ 



We now prove Proposition 13. II by induction on sdim(S). If it is zero, then 

J(E) = Z(E)U E x H(£), 

and we are done by Lemma 13.61 Now assume that it is positive and we have proved 
the proposition when sdim(.E') is smaller. Then Proposition 12.11 implies that every 
T G D-™(J(E)) is supported in (Z(E)U E ) K H(£), and then T = by Lemma ESI 
This finishes the proof. 



4. Reduction within the null cone 

In this section, we treat the case of symplectic groups only. Througout this section, 
assume that e = — 1, A is a field, and that the involution r on A is trivial. 
Let 

M E = Af D Mi D ■ ■ ■ D Mr = {0} D A^+i = 
be a filtration of Me by its closed subsets so that each difference 

O i :=M l \M l+ll 0<*<r, 

is a U(.£7)-orbit. The goal of this section is to prove the following 

Proposition 4.1. If every distribution in D~^(u(E) x E) is supported in Mi x E, 
for some fixed < i < r, then every distribution in D^(u(E) x E) is supported in 
M i+ i x E. 
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Without loss of generality, in the remaining part of this section we further assume 
that A = k. Fix a point e of 0j. Recall that Oi is said to be distinguished if e 
commutes with no nonzero semisimple element in u(E) (c.f. |CM93t Section 8.2]). 
This definition is independent of the choice of e. We first treat the case when Oi is 
not distinguished. View u(E) as a quadratic space over k under the trace form 

(x,y) u (E) ■= tr(xy). 
Recall the following elementary fact (c.f., [SZ0821 Lemma 4.2]). 

Lemma 4.2. If Oi is not distinguished, then there is a non-isotropic vector in u(E) 
which is perpendicular to the tangent space T e (Oi) C u(E). 

Recall the action fll]) of tj(E) on u(E) and E. Write E' := E as an e-Hermitian 
A-module, but equipped with the action of IJ(E) given by 

(g,5).u := gu. 

Let tl(E) act on u(E) x E' diagonally. Define a non-degenerate U(.E)-invariant 
bilinear map 

(,),•: (u(E) x E) x (u(E) x £') -> k 

by 

((x, u), (x 1 , u'))y := (x, x') u (e) + (u, u') E - 
Fix a nontrivial character ip of k. As in the appendix, for every distribution T e 
D-°°(u(E) x E), define its Fourier transform f G Cr°°(u(E) x E') by 

T{uj) := T(u), u e Dq(u(E) x E'), 
where u G C^°(u(i?) x E) is given by 



w0'):=/ H(j,fh)du(f), jeu(E)xE. 

Ju(E)xE' 

Lemma 4.3. Assume that Oi is not distinguished. Let T G D~°°(u(E) x E). If T 
is supported in Mi x E, and its Fourier transform T G C~°°(u(E) x E') is supported 
in the null cone 

(11) {{x, u) G u(E) x E' | tr(x 2 ) = 0}, 

then T is supported in A/i+i x E 1 . 

Proof. This is a direct consequence of Lemma T4. 21 and Theorem lA.ll of the appendix. 

□ 



Lemma 4.4. Proposition 4-1 holds when Oi is not distinguished. 
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Proof. Let T G D~^(u(E) x E). Then by assumption it is supported in Mi x It 
is clear that the Fourier transform maps D~™(u(E) x E) into C^°(u(i?) x E'). By 
noting that — 1 G XJ(E), we find that the space C^(\x(E) x E') is identical with the 

space C7^^°(u(-E') x E). Apply the assumption to T, we find that T is supported in 
Mi x E' , which is contained in the null cone (fTTj) . We finish the proof by Lemma I4T51 

□ 

Now we treat the case when Oi is distinguished. For all v G E, put 

4> v (u) := (u,v) E v, uEE. 
One easily checks that <p v G u(-E'). For all o E Oi, put 

£(o) := { w G £7 | 0, G [u(E),o]}. 

Lemma 4.5. // even/ distribution in D~™(u(E) x £7) is supported in Mi x £7 ; t/ien 
i/ie support of every distribution in D~™(u(E) x E) is contained in 

(M+i x U □ {o} x £7( ). 

Proo/. We follow the method of |AGRS07j . Let T G £^°(u(E) x E) and (o,v) G 
Oj x E be a point in the support of T. It suffices to prove that v G E{o). 
For every t G k, define a homeomorphism 

7ft : u(£?) x E -> u(£) x £, 

i-> (x + t(j) u ,u), 

which is checked to be U(.E)-intertwining. Therefore 

WJe]}->(£)x£). 

Since (o, f) is in the support of T, r) t (o,v) is in the support of (r]t)*T. Therefore the 
assumption implies that 

(12) Vt(o,v) = {o + t<f> v ,v) G Mi x E. 

As Oi is open in Mi, (jl2l) implies that 

0, G T (Oi) = [u(£),o]. 

□ 

Extend e to a standard triple h, e, f in u(E), i.e., the k-linear map from ^(k) to 
u(E) specified by 



" 1 


i-> h, 


" 


1 " 




" 


" 


-1 








i-> e, 







1 
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is a Lie algebra homomorphism. Existence of such an extension is known as Jacobson- 
Morozov Theorem. Using this homomorphism, we view E as an s[2(k)-module with 
an invariant symplectic form. In the remaining part of this section assume that 
Oi is distinguished. By the classification of distinguished nilpotent orbits ( |CM93| 
Theorem 8.2.14]), we know that E has an orthogonal decomposition 

E = E 1 © E 2 © • • ■ © E s , s > 0, 

where all E^s are irreducible sl2(k)-submodules, with pairwise different even dimen- 
sions. Denote by E + and E~ the subspaces of E spanned by eigenvectors of h with 
positive and negative eigenvalues, respectively. Then 

E = E + © E~ 

is a complete polarization of E. 

We omit the proof of the following elementary lemma (c.f. |SZ082[ Lemma 3.4]). 



Lemma 4.6. One has that 

E(e) C E + . 

Fix a Haar measure du' on E' . For any t.d. space M, we define the partial Fourier 
transform 

J= E : D-°°(M x E) -»■ D-°°(M x E') 

by 

Fe(T)(<Pm ® <p') ■= T(<p M © V M G Cg°(M), ^ G Cg° (E'), 
where G C^(E) is given by 



tp'(u) := / 'ip((u,u) E )ip , (u , )du'. 
Je> 

For every o G Oj, write £"(o) := -E(o), viewed as a subset of 



Lemma 4.7. Lei T G D~°°{E). If T is supported in E(e), and Fe{T) G D~°°{E') 
is supported in E'(e), then T is a scalar multiple of a Haar measure of E + . 

Proof. Since Te(T) is supported in E'(e), T is invariant under translations by ele- 
ments of 

{u G E | (u,u') E = 0, u'eE'(e)}. 
Lemma then implies that T is supported in E + and is invariant under translations 
by E + . This proves the lemma. □ 

Denote by \J(E, e) the stabilizer of e G u(E) in U(£ l ), and by XE,e the restriction 
of X E to U(£,e). 
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Lemma 4.8. Let T G D ™ e (E). IfT is supported in E{e), and J-e(T) is supported 
in E'(e), then T = 0. 

Proof. By Lemma \A.7\ T is a scalar multiple of a Haar measure of E + . Note that all 
eigenvalues of h on E are odd integers. Let g : E — > E be the linear map which is the 
scalar multiplication by (— l) n on the h-eigenspace with eigenvalue 2n + 1, n G Z. It 
is clear that (g, —1) G U(£ l , e), and leaves the Haar measure of i? + -invariant. This 
finishes the proof. □ 

Fix a positive U(£')-invariant measure do on (9, (which always exists), and a Haar 
measure dg on IJ(E). Define a submersion 



where du is any Haar mesure on E, f G C~°°(Oi x £7), and is the usual pull 

back of a generalized function. By Frobenious reciprocity (c.f. [Be84| Section 1.5]), 
there is a well defined linear isomorphism 



p e : XJ(E) xE Cj x E, 
(g,v) i-)> #.(e,w), 



and define the pull back 



: ZT 00 ^ x £) Zr°°(U(£) x £) 
fdo®du i — ^ pl(f)dg®du, 




r e : x £) ^> £-(£), 




and a linear isomorphism 



The routine verification of the following lemma is left to the reader. 
Lemma 4.9. The diagram 



D^iOi x E) D--(E) 



D^iPi x E') 



commutes. 
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Now we are ready to prove Proposition 14.11 when 0, is distinguished. Let T e 
D~™(J\fi x E). Then Lemma S3] implies that 

r e (T| 0iX£ ) e D~™ e (E) is supported in E{e). 
Similarly, since Fe(T) E D~™(Ni x we have 

r' e {F E {T)\ 0iXE >) e D~™ e {E') is supported in E'{e). 
Now Lemma [4.81 and Lemma [4.91 implies that 

r e (T\ 0iXE ) = 0, 
which implies that T\o t xE = 0. This finishes the proof. 

5. Proof of the theorems 

We return to the general case. Recall that A is a commutative involutive algebra, 
and E is an e-Hermitian A-module. 

Theorem 5.1. One has that 

(14) C£°(u(£) x E) = 0. 

Proof. Write s := sdim(£') > 0, and assume that the theorem is proved when s is 
smaller. Without loss of generality, we assume that A is simple (c.f. [SZ082, Lemma 
6.1]). When e = 1, (114"]) is proved in |AGRS07] . which is the infinitesimal version 
of the main result of that paper. When e = — 1 and r is nontrivial on A, take an 
element 

c A e u(A) n A x . 

Then (E,ca(, )e) is a —e-Hermitian A-module, and ffT4"j) reduces to the case when 
e = 1. 

Now to finish the proof, we further assume that e = — 1 and r is trivial on A. 
Then A is a field. By Proposition I2.2[ every generalized function in C^{\x{E) x i£) 
is supported in 

( d (E)+Af E )xE = Af E xE, 
and it has to vanish by Proposition 14.11 □ 

Theorem D is now a consequence of Theorem 15.11 and Proposition 13.11 

The following Lemma is well known. It is also a direct consequence of Theorem 

D. 
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Lemma 5.2. For every f G C^ 00 (U(£')), if 

figxg- 1 ) = f{x) for all g G U(£), 

then 

figx-'g- 1 ) = f(x) for all g G XJ(E) \ XJ(E). 

Theorem C is a consequence of Theorem D, Lemma \5. 21 and the following version 
of the Gelfand-Kazhdan criterion. 

Lemma 5.3. Let G be a t.d. group with a closed subgroup S . Let a be a continuous 
anti- automorphism of G such that o~(S) = S . Assume that for every generalized 
function f on G or on S , the condition 

f(sxs~) = f(x) for all s G S 

implies that 

f(x°) = /On). 

Then for all irreducible admissible smooth representation ttq of G, and its of S, we 
have 

dimHoHl5(7TG, VTg) < 1. 

Proof. This is proved for real reductive groups in |SZ082l Proposition 7.1]. The same 
proof works here. We sketch a proof for the convenience of the reader. 

Denote by A(S') the diagonal subgroup S of G x S. The assumption on G implies 
that every bi-A(S') invariant generalized function on G x S is o x er-invariant. Then 
the usual Gelfand-Kazhdan criterion (c.f. [SZ0811 Theorem 2.3]) implies that 

(15) dimHom,s(7r G ® TCg, C) • dimHom^vr^ <g> ir s , C) < I. 

Here and henceforth, " v " stands for the contragredient of an admissible smooth 
representation. 

Denote by a 1 the automorphism g i— > a{g~ l ). By considering characters of irre- 
ducible representations (which are conjugation invariant generalized functions on the 
groups), the assumption implies that 

n v G = ir^ and n s = (n^' . 

Here ttq is the representation of G which has the same underlying space as that of 
tig, and whose action is given by g \-> ira(cr'(g)). The representation (ng) a is defined 
similarly. Therefore the two factors in (Tl5l) are equal to each other, and consequently, 

dimHom5(7r G < <E> vr^, C) < 1, 

or the same, 

dimHonis(7r G , tc s ) < 1. 

□ 
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To prove Theorem B, we assume that e = — 1, and 

{(k x k, the nontrivial automophism), 
(a quadratic field extension of k, the nontrivial automophism) , or 
(k, the trivial automophism). 

Then U{E) is GL(n), U(n), or Sp(2n), respectively, with 2n = dimk(-E'). Write 
E k := E, viewed as a symplectic k-vector space under the form 

(u,v) Ek := tr A/k ((u,v) E ). 

Denote by Sp(-Bk) the metaplectic cover of the symplectic group Sp(-Bk)- It induces 
a double cover U{E) of U{E) C Sp(i?k)- For any non-trivial character ip of k, denote 
by aty the corresponding smooth oscillator representation of 

Sp(E k ) x H(£ k ). 

Up to isomorphism, this is the only genuine smooth representation which, as a rep- 
resentation of H(E k ), is irreducible and has central character ip. 

Now it suffices to show the following lemma. This is actually known (see [AP06, 
Page 222]). We provide a proof for the sake of completeness. 

Lemma 5.4. With the notation as above, for every genuine irreducible admissible 
smooth representation of U(E), the tensor product <g) lo^ is an irreducible 
admissible smooth representation ofU(E) x H(E k ). 

Proof. The smoothness and admissibility are clear. We prove that 7r^ eg) is irre- 
ducible as a smooth representation of U{E) x H(E k ). The space 

H := Hom H (B t )(w^, 7^ <g> u^) 

is a smooth representation of U(E) under the action 

(~g4)(v) := g.^ig-'-v)), 

where 

g e U(E), cj) e H , v G uty, 

and g is the image of g under the quotient map ~U(E) — > U{E). Let 7Tj be a nonzero 
XJ(E) k H(i?k)-subrepresentation of <S> uj^, then 

Hom H (B k )(o;^,7rj) 
is a nonzero U(£')-subrepresentation of H . Since the linear map 

7T|j H , t)H>t)(8l(-) 

is bijective and U(i?)-intertwining, Ho is irreducible. Therefore 

HoniH(B)(w^,7Tj) = H , 
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and consequently, 7Tj = <g) ay,. 

□ 



Appendix A. An uncertainty theorem for distributions with supports 

Let k be a non-archimedean local field of characteristic zero. Fix a non-trivial 
character ip of k. Let E and F be two finite-dimensional k-vector spaces which are 
dual to each other, i.e., a non-degenerate bilinear map 

(, ) : E x F ^k 

is given. The Fourier transform 

D™(F) C?(E) 
uj i — y Cj 

is the linear isomorphism given by 

u>(x) := J tp({x,y))duj(y), x E E. 

For every T G D~°°(E), its Fourier transform T G C^°°(F) is given by 

T(u):=T(u>), coeD^(F). 

For every subset X of E, a point i 6 I is said to be regular if there is an open 
neighborhood U of x in E such that U (1 X is a closed locally analytic submanifold 
of U. In this case, the tangent space T X (X) C E is defined as usual. We define the 
conormal space to be 

N;(X) :={veF\ (u, v) = 0, u G T X (X)}. 

The uncertainty principle says that a distribution and its Fourier transform can 
not be simultaneously arbitrarily concentrated. The purpose of this appendix is to 
prove the following theorem, which is a form of the uncertainty principle. 

Theorem A.l. Let x be a regular point in a close subset X of E. Let f : F — > k be 
a polynomial function of degree d > 1. Let T G D~°°(E) be a distribution supported 
in X, with its Fourier transform T G C~°°(F) supported in the zero locus of f . 
If fd take nonzero values at some points o/N*(X) 7 then T vanishes on some open 
neighborhood of x in E, where fd is the homogeneous component of f of degree d. 
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Remark: The archimedean analog of Theorem IA.1I also holds. This is a direct 
consequence of [JSZ08, Lemma 2.2]. 

Fix a non-archimedean multiplicative norm 

| • | k : k -> [0, +oo) 

which defines the topology of k. Also fix a non-archimedean norm (multiplicative 
with respect to | • |k) 

| • \ F : F -» [0,+oo), 

which automatically defines the topology of F. 

Let / and fd be as in Theorem IA.1) and denote by Zf C F the zero locus of /. 
Write fa '■= f — fdi which is a polynomial function of degree < d — 1. Then 

(16) |/o(3/)|k = o(|3/|!0, as|y| F ^+oo. 

Lemma A. 2. Let Vi and Vi be two compact open subsets of F. If fd has no zero in 
V2, then 

(Z f + K) n \V 2 = 
for all A G k x with |A|k sufficiently large. 

Proof. Take a positive number c so that 

(17) \f d (y)\ k >c\y\ d F , for ally Gk x V 2 . 
It is easy to see that 

(18) max \ f d (y + v) - f d (y)\ k = o(\y\ d F ), as \y\ F +00. 

If y G Z/, then 

(19) l/d(y + «)| k 

< max{ \f d (y)\k, \fd(y + v)-f d (y)\\) 
= max{ |/o(2/) |k, + - /d(y)|k}- 

The inequalities (fT6|) . (flBl and (|T9|) implies that 

(20) \fd(v)W=o(\y\j), as y e Z f + V u and \y\ F +00. 
The lemma then follows by comparing (ITT)) and (120)) . 

□ 

Recall the following 
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Definition A.3. (c.f., JHe85l Section 2}) A distribution T G D~°°(E) is said to be 
smooth at a point (x,y) G E x F if there is a compact open neighborhood U of x, and 
a compact open neighborhood V of y such that the Fourier transform \\jT vanishes 
on XV for all A G k x with |A|k sufficiently large. Here \\j stands for the characteristic 
function of U . The wave front set ofTatxEEis defined to be 

WF X (T) := {y G F \ T is not smooth at (x,y)}. 

Clearly, the wave front set WF X (T) is closed in F and is stable under multiplica- 
tions by k x . 

Lemma A. 4. If the Fourier transform T of a distribution T G D~°°(E) is supported 
in Zf, then for every x G E, the wave front set WF X (T) is contained in the zero 
locus of f d . 

Proof. Let y G F be a vector so that fd{y) ^ 0. We need to show that T is smooth 
at (x,y). Take an arbitrary compact open neighborhood U of x, and an arbitrary 
compact open neighborhood V of y so that fd has no zero in V. We claim that \\jT 
vanishes on XV for all A G k x with |A|k sufficiently large. The lemma is a consequence 
of this claim. 

Note that l^T is a finite linear combination of generalized functions of the form 

(lv! dy) * T, V\ is a compact open subset of F. 

Here dy is a fixed Haar measure on F. The support of the convolution [ly 1 dy) * T 
is contained in Zj + V\. Therefore the claim follows from Lemma [A. 21 □ 

Lemma A. 5. If a distribution T G D~°°(E) is supported in a closed subset X of 
E, and x G X is a regular point, then the wave front set WF X (T) is invariant under 
translations by elements ofN*(X) C F. 

Proof. This is proved in [Ai08[ Therorem 4.1.2]. We indicate the main steps. 

Step 1. When we replace a distribution by a translation of it, the wave front set 
does not change. Therefore we may assume that x — 0. 

Step 2. Let (p : E — Y E be a locally analytic diffeomorphism which sends to 
and induces the identity map on the tangent space at 0. When we replace T by 
its pushing forward <p*(T), the wave front set WF (T) does not change. Therefore 
we may assume that X D U = Eq D U, for some subspace E of E, and some open 
neighborhood U of 0. 

Step 3. When we replace T by a distribution which coincides with T on an open 
neighborhood of 0, the wave front set WF (T) does not change. Therefore we may 
assume that X = E . 
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Step 4. Assume that T is supported in E . Then T is invariant under translations 

by 

N;(X) = Ei :={veF\ (u,v) = 0, u G £ }, 
which implies that the same holds for WFo(T). □ 

Now we are ready to prove Theorem IA.1I It is clear that T vanishes on some 
open neighborhood of x if and only if ^ WF X (T). If G WF X (T), then Lemma 
\KE implies that N*(X) C WF X (T). Now Lemma El further implies that N*(X) is 
contained in the zero locus of fd, which contradicts the assumption of the theorem. 
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